Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee $\&$ Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex)\footnote{source code: https://github.com/wh1992v/ggp-rl}, to allow comparison to Banerjee et al.. We find that Q-learning converges to a high win rate in GGP. For the $\epsilon$-greedy strategy, we propose a first enhancement, the dynamic $\epsilon$ algorithm. In addition, inspired by (Gelly $\&$ Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Comment: arXiv admin note: substantial text overlap with arXiv:1802.0594

Agent cognition through micro-simulations: Adaptive and tunable intelligence with NetLogo LevelSpace

We present a method of endowing agents in an agent-based model (ABM) with sophisticated cognitive capabilities and a naturally tunable level of intelligence. Often, ABMs use random behavior or greedy algorithms for maximizing objectives (such as a predator always chasing after the closest prey). However, random behavior is too simplistic in many circumstances and greedy algorithms, as well as classic AI planning techniques, can be brittle in the context of the unpredictable and emergent situations in which agents may find themselves. Our method, called agent-centric Monte Carlo cognition (ACMCC), centers around using a separate agent-based model to represent the agents' cognition. This model is then used by the agents in the primary model to predict the outcomes of their actions, and thus guide their behavior. To that end, we have implemented our method in the NetLogo agent-based modeling platform, using the recently released LevelSpace extension, which we developed to allow NetLogo models to interact with other NetLogo models. As an illustrative example, we extend the Wolf Sheep Predation model (included with NetLogo) by using ACMCC to guide animal behavior, and analyze the impact on agent performance and model dynamics. We find that ACMCC provides a reliable and understandable method of controlling agent intelligence, and has a large impact on agent performance and model dynamics even at low settings.Comment: Model source code available here: https://github.com/qiemem/Wolf-Sheep-Predation-Micro-Sims, In: Unifying Themes in Complex Systems IX. ICCS 2018. Springer Proceedings in Complexity. Springer, Cha

Preference-Based Monte Carlo Tree Search

Monte Carlo tree search (MCTS) is a popular choice for solving sequential anytime problems. However, it depends on a numeric feedback signal, which can be difficult to define. Real-time MCTS is a variant which may only rarely encounter states with an explicit, extrinsic reward. To deal with such cases, the experimenter has to supply an additional numeric feedback signal in the form of a heuristic, which intrinsically guides the agent. Recent work has shown evidence that in different areas the underlying structure is ordinal and not numerical. Hence erroneous and biased heuristics are inevitable, especially in such domains. In this paper, we propose a MCTS variant which only depends on qualitative feedback, and therefore opens up new applications for MCTS. We also find indications that translating absolute into ordinal feedback may be beneficial. Using a puzzle domain, we show that our preference-based MCTS variant, wich only receives qualitative feedback, is able to reach a performance level comparable to a regular MCTS baseline, which obtains quantitative feedback.Comment: To be publishe

The association of cold weather and all-cause and cause-specific mortality in the island of Ireland between 1984 and 2007

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.This article has been made available through the Brunel Open Access Publishing Fund.Background This study aimed to assess the relationship between cold temperature and daily mortality in the Republic of Ireland (ROI) and Northern Ireland (NI), and to explore any differences in the population responses between the two jurisdictions. Methods A time-stratified case-crossover approach was used to examine this relationship in two adult national populations, between 1984 and 2007. Daily mortality risk was examined in association with exposure to daily maximum temperatures on the same day and up to 6 weeks preceding death, during the winter (December-February) and cold period (October-March), using distributed lag models. Model stratification by age and gender assessed for modification of the cold weather-mortality relationship. Results In the ROI, the impact of cold weather in winter persisted up to 35 days, with a cumulative mortality increase for all-causes of 6.4% (95%CI=4.8%-7.9%) in relation to every 1oC drop in daily maximum temperature, similar increases for cardiovascular disease (CVD) and stroke, and twice as much for respiratory causes. In NI, these associations were less pronounced for CVD causes, and overall extended up to 28 days. Effects of cold weather on mortality increased with age in both jurisdictions, and some suggestive gender differences were observed. Conclusions The study findings indicated strong cold weather-mortality associations in the island of Ireland; these effects were less persistent, and for CVD mortality, smaller in NI than in the ROI. Together with suggestive differences in associations by age and gender between the two Irish jurisdictions, the findings suggest potential contribution of underlying societal differences, and require further exploration. The evidence provided here will hope to contribute to the current efforts to modify fuel policy and reduce winter mortality in Ireland

Urban women's socioeconomic status, health service needs and utilization in the four weeks after postpartum hospital discharge: findings of a Canadian cross-sectional survey

<p>Abstract</p> <p>Background</p> <p>Postpartum women who experience socioeconomic disadvantage are at higher risk for poor health outcomes than more advantaged postpartum women, and may benefit from access to community based postpartum health services. This study examined socioeconomically disadvantaged (SED) postpartum women's health, and health service needs and utilization patterns in the first four weeks post hospital discharge, and compared them to more socioeconomically advantaged (SEA) postpartum women's health, health service needs and utilization patterns.</p> <p>Methods</p> <p>Data collected as part of a large Ontario cross-sectional mother-infant survey were analyzed. Women (N = 1000) who had uncomplicated vaginal births of single 'at-term' infants at four hospitals in two large southern Ontario, Canada cities were stratified into SED and SEA groups based on income, social support and a universally administered hospital postpartum risk screen. Participants completed a self-administered questionnaire before hospital discharge and a telephone interview four weeks after discharge. Main outcome measures were self-reported health status, symptoms of postpartum depression, postpartum service needs and health service use.</p> <p>Results</p> <p>When compared to the SEA women, the SED women were more likely to be discharged from hospital within the first 24 hours after giving birth [OR 1.49, 95% CI (1.01–2.18)], less likely to report very good or excellent health [OR 0.48, 95% CI (0.35–0.67)], and had higher rates of symptoms of postpartum depression [OR 2.7, 95% CI(1.64–4.4)]. No differences were found between groups in relation to self reported need for and ability to access services for physical and mental health needs, or in use of physicians, walk-in clinics and emergency departments. The SED group were more likely to accept public health nurse home visits [OR 2.24, 95% CI(1.47–3.40)].</p> <p>Conclusion</p> <p>Although SED women experienced poorer mental and overall health they reported similar health service needs and utilization patterns to more SEA women. The results can assist policy makers, health service planners and providers to develop and implement necessary and accessible services. Further research is needed to evaluate SED postpartum women's health service needs and barriers to service use.</p

Assessing the Potential of Classical Q-learning in General Game Playing

After the recent groundbreaking results of AlphaGo and AlphaZero, we have seen strong interests in deep reinforcement learning and artificial general intelligence (AGI) in game playing. However, deep learning is resource-intensive and the theory is not yet well developed. For small games, simple classical table-based Q-learning might still be the algorithm of choice. General Game Playing (GGP) provides a good testbed for reinforcement learning to research AGI. Q-learning is one of the canonical reinforcement learning methods, and has been used by (Banerjee & Stone, IJCAI 2007) in GGP. In this paper we implement Q-learning in GGP for three small-board games (Tic-Tac-Toe, Connect Four, Hex), to allow comparison to Banerjee et al. We find that Q-learning converges to a high win rate in GGP. For the ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ-greedy strategy, we propose a first enhancement, the dynamic ϵ" role="presentation" style="display: inline-table; line-height: normal; letter-spacing: normal; word-spacing: normal; overflow-wrap: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0px; min-height: 0px; border-width: 0px; border-style: initial; position: relative;">ϵ algorithm. In addition, inspired by (Gelly & Silver, ICML 2007) we combine online search (Monte Carlo Search) to enhance offline learning, and propose QM-learning for GGP. Both enhancements improve the performance of classical Q-learning. In this work, GGP allows us to show, if augmented by appropriate enhancements, that classical table-based Q-learning can perform well in small games.Computer Systems, Imagery and Medi

Bryozoans are Major Modern Builders of South Atlantic Oddly Shaped Reefs

Supplementary information accompanies this paper at https://doi.org/10.1038/s41598-018-27961-6.In major modern reef regions, either in the Indo-Pacific or the Caribbean, scleractinian corals are described as the main reef framework builders, often associated with crustose coralline algae. We used underwater cores to investigate Late Holocene reef growth and characterise the main framework builders in the Abrolhos Shelf, the largest and richest modern tropical reef complex in the South Western Atlantic, a scientifically underexplored reef province. Rather than a typical coralgal reef, our results show a complex framework building system dominated by bryozoans. Bryozoans were major components in all cores and age intervals (2,000 yrs BP), accounting for up to 44% of the reef framework, while crustose coralline algae and coral accounted for less than 28 and 23%, respectively. Reef accretion rates varied from 2.7 to 0.9 mm yr−1, which are similar to typical coralgal reefs. Bryozoan functional groups encompassed 20 taxa and Celleporaria atlantica (Busk, 1884) dominated the framework at all cores. While the prevalent mesotrophic conditions may have driven suspensionfeeders’ dominance over photoautotrophs and mixotrophs, we propose that a combination of historical factors with the low storm-disturbance regime of the tropical South Atlantic also contributed to the region’s low diversity, and underlies the unique mushroom shape of the Abrolhos pinnacles.We thank CNPq/FAPES-Sisbiota/PELD, CAPES/IODP, CAPES/Ciências do Mar, and ANP/Brasoil for long term project funding. We also thank ICMBio for research permits and field logistic support, and Conservation International for providing and authorizing the use of the IKONOS image. JMW and JCB are International Visiting Researcher at UFES and JBRJ, supported by the Science Without Borders program. Zá Cajueiro provided invaluable field support and Ronaldo Francini, Carlos Janovitch and Lucio Engler helped in the drilling operations. This is a contribution from the Rede Abrolhos (abrolhos.org)

Firsthand Experience and The Subsequent Role of Reflected Knowledge in Cultivating Trust in Global Collaboration

While scholars contend that firsthand experience - time spent onsite observing the people, places, and norms of a distant locale - is crucial in globally distributed collaboration, how such experience actually affects interpersonal dynamics is poorly understood. Based on 47 semistructured interviews and 140 survey responses in a global chemical company, this paper explores the effects of firsthand experience on intersite trust. We find firsthand experience leads not just to direct knowledge of the other, but also knowledge of the self as seen through the eyes of the other - what we call “reflected knowledge”. Reflected and direct knowledge, in turn, affect trust through identification, adaptation, and reduced misunderstandings

Maternal Use of Antibiotics, Hospitalisation for Infection during Pregnancy, and Risk of Childhood Epilepsy: A Population-Based Cohort Study

BACKGROUND: Maternal infection during pregnancy may be a risk factor for epilepsy in offspring. Use of antibiotics is a valid marker of infection. METHODOLOGY/PRINCIPAL FINDINGS: To examine the relationship between maternal infection during pregnancy and risk of childhood epilepsy we conducted a historical cohort study of singletons born in northern Denmark from 1998 through 2008 who survived ≥29 days. We used population-based medical databases to ascertain maternal use of antibiotics or hospital contacts with infection during pregnancy, as well as first-time hospital contacts with a diagnosis of epilepsy among offspring. We compared incidence rates (IR) of epilepsy among children of mothers with and without infection during pregnancy. We examined the outcome according to trimester of exposure, type of antibiotic, and total number of prescriptions, using Poisson regression to estimate incidence rate ratios (IRRs) while adjusting for covariates. Among 191,383 children in the cohort, 948 (0.5%) were hospitalised or had an outpatient visit for epilepsy during follow-up, yielding an IR of 91 per 100 000 person-years (PY). The five-year cumulative incidence of epilepsy was 4.5 per 1000 children. Among children exposed prenatally to maternal infection, the IR was 117 per 100,000 PY, with an adjusted IRR of 1.40 (95% confidence interval (CI): 1.22-1.61), compared with unexposed children. The association was unaffected by trimester of exposure, antibiotic type, or prescription count. CONCLUSIONS/SIGNIFICANCE: Prenatal exposure to maternal infection is associated with an increased risk of epilepsy in childhood. The similarity of estimates across types of antibiotics suggests that processes common to all infections underlie this outcome, rather than specific pathogens or drugs

Isomer Spectroscopy of Neutron-rich 165,167Tb

Open Access JournalWe present information on the excited states in the prolate-deformed, neutron-rich nuclei 165;167Tb100;102. The nuclei of interest were synthesized following in-flight fission of a 345 MeV per nucleon 238U primary beam on a 2 mm 9Be target at the Radioactive Ion-Beam Factory (RIBF), RIKEN, Japan. The exotic nuclei were separated and identified event-by-event using the BigRIPS separator, with discrete energy gamma-ray decays from isomeric states with half-lives in the _s regime measured using the EURICA gamma-ray spectrometer. Metastable-state decays are identified in 165Tb and 167Tb and interpreted as arising from hindered E1 decay from the 7/2-[523] single quasi-proton Nilsson configuration to rotational states built on the 3/2-[411] single quasi-proton ground state. These data correspond to the first spectroscopic information in the heaviest, odd-A terbium isotopes reported to date and provide information on proton Nilsson configurations which reside close to the Fermi surface as the 170Dy doubly-midshell nucleus is approached.postprin